Content-modification system with unscheduling feature

ABSTRACT

In one aspect, a method includes a content-presentation device determining first and second intervals during which presentation of received content is modifiable, and transmitting first and second requests for first and second supplemental content. A link to content in a playlist may be received, the playlist including first supplemental content followed immediately by second supplemental content. The link may be used to retrieve the first supplemental content, which may be presented in a first modification operation during the first interval. Before completing the first modification operation, the content-presentation device may use the link to start retrieving the second supplemental content for a second modification operation scheduled immediately following the first modification operation, and may determine that while less than a threshold portion of the first modification operation remains incomplete, retrieval of the second supplemental content is also incomplete. The content-presentation device may then unschedule the second content-modification operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application Ser. No. 62/858,873, filed on Jun. 7,2019, which is incorporated herein in its entirety by reference.

USAGE AND TERMINOLOGY

In this disclosure, unless otherwise specified and/or unless theparticular context clearly dictates otherwise, the terms “a” or “an”mean at least one, and the term “the” means the at least one.

SUMMARY

In one aspect, a method includes: while receiving content forpresentation, a content-presentation device determining an upcomingfirst temporal interval and an immediately following second temporalinterval during both of which the presentation of the received contentis modifiable; transmitting, by the content-presentation device, a firstrequest for first supplemental content; transmitting, by thecontent-presentation device, a second request for second supplementalcontent; receiving, by the content-presentation device, a link thatpoints to a set of content items arranged in a playlist, wherein whenthe first request is transmitted, the playlist includes the requestedfirst supplemental content, and at subsequent time after the firstrequest is transmitted the playlist is modified to also include therequested second supplemental content positioned immediately after thefirst supplemental content; using, by the content-presentation device,the received link to retrieve the first supplemental content; using, bythe content-presentation device, the retrieved first supplementalcontent in a first content-modification operation to modify presentationduring the first temporal interval of the received content; prior tocompletion of the first content-modification operation: (i) using, bythe content-presentation device, the received link to start retrievingthe second supplemental content for upcoming use in a secondcontent-modification operation scheduled immediately following the firstcontent-modification operation in order to modify presentation duringthe second temporal interval of the received content, and (ii) making adetermination, by the content-presentation device, that while a lessthan threshold portion of the first content-modification operationremains to be completed, the content-presentation device has notcompleted retrieving the second supplemental content; and responsive tothe determination, unscheduling, by the content-presentation device, thesecond content-modification operation.

In another aspect, a content-presentation device includes acontent-presentation element configured for presenting content receivedby the content-presentation device; one or more processors; and anon-transitory computer-readable storage medium having stored thereonprogram instructions that, upon execution by the one or processors,cause the content-presentation device to carry out operations including:while receiving content for presentation on the content-presentationelement, determining an upcoming first temporal interval and animmediately following second temporal interval during both of which thepresentation of the received content is modifiable; transmitting a firstrequest for first supplemental content; transmitting a second requestfor second supplemental content; receiving a link that points to a setof content items arranged in a playlist, wherein when the first requestis transmitted, the playlist includes the requested first supplementalcontent, and at subsequent time after the first request is transmittedthe playlist is modified to also include the requested secondsupplemental content positioned immediately after the first supplementalcontent; using the received link to retrieve the first supplementalcontent; using the retrieved first supplemental content in a firstcontent-modification operation to modify presentation during the firsttemporal interval of the received content; prior to completion of thefirst content-modification operation: (i) using the received link tostart retrieving the second supplemental content for upcoming use in asecond content-modification operation scheduled immediately followingthe first content-modification operation in order to modify presentationduring the second temporal interval of the received content, and (ii)making a determination that while less than a threshold portion of thefirst content-modification operation remains to be completed, retrievingthe second supplemental content has not yet completed; and responsive tothe determination, unscheduling the second content-modificationoperation.

In another aspect, a non-transitory computer-readable storage mediumhaving stored thereon program instructions that, upon execution by oneor more processors of a content-presentation device, cause thecontent-presentation device to carry out operations. The operationsinclude: while receiving content for presentation determining anupcoming first temporal interval and an immediately following secondtemporal interval during both of which the presentation of the receivedcontent is modifiable; transmitting a first request for firstsupplemental content; transmitting a second request for secondsupplemental content; receiving a link that points to a set of contentitems arranged in a playlist, wherein when the first request istransmitted, the playlist includes the requested first supplementalcontent, and at subsequent time after the first request is transmittedthe playlist is modified to also include the requested secondsupplemental content positioned immediately after the first supplementalcontent; using the received link to retrieve the first supplementalcontent; using the retrieved first supplemental content in a firstcontent-modification operation to modify presentation during the firsttemporal interval of the received content; prior to completion of thefirst content-modification operation: (i) using the received link tostart retrieving the second supplemental content for upcoming use in asecond content-modification operation scheduled immediately followingthe first content-modification operation in order to modify presentationduring the second temporal interval of the received content, and (ii)making a determination that while less than a threshold portion of thefirst content-modification operation remains to be completed, retrievingthe second supplemental content has not yet completed; and responsive tothe determination, unscheduling the second content-modificationoperation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of an example content-modificationsystem in which various described principles can be implemented.

FIG. 2 is a simplified block diagram of an example computing system inwhich various described principles can be implemented.

FIG. 3 is a diagram of example linear sequences of content and relatedconcepts.

FIGS. 4A, 4B, 4C, 4D, 4E, and 4F collectively make up a table showingexample time-periods and corresponding operations that can be performedin connection with the example content-modification system.

FIG. 5A is an illustration of operations relating to stitching ofcontent, in accordance with example embodiments.

FIG. 5B in an illustration of operations relating to unscheduling of acontent-modification operation, in accordance with example embodiments.

FIG. 6 is a flow chart of an example method.

DETAILED DESCRIPTION I. Overview

To deliver and present content to end-users, a content provider cantransmit the content to one or more content-distribution systems, eachof which can in turn transmit the content to one or more respectivecontent-presentation devices to be output for presentation to respectiveend-users. Such a hierarchical arrangement can facilitate convenient,widespread distribution of content.

By way of example, in order for a video content provider to delivervideo content to end-users throughout the United States, the videocontent provider can transmit the video content by satellite or anothermedium to content-distribution systems that serve respective designatedmarket areas (DMAs) within the United States. Each suchcontent-distribution system can therefore receive the national satellitefeed carrying the video content and can transmit the video content totelevision sets and/or set-top boxes in the content-distributionsystem's DMA, such that the video content can be output for presentationto respective end-users in that DMA. In practice, thesecontent-distribution systems and their means of transmission tocontent-presentation devices can take various forms. For instance, acontent-distribution system can be associated with a cable-televisionprovider and can transmit video content to content-presentation devicesof end-users who are cable-television subscribers through hybridfiber/coaxial cable connections.

As such, in various scenarios, a content-distribution system cantransmit content to a content-presentation device, which can receive andoutput the content for presentation to an end-user. In some situations,even though the content-presentation device receives content from thecontent-distribution system, it can be desirable for thecontent-presentation device to perform a content-modification operationso that the content-presentation device can output for presentationalternative content instead of at least a portion of that receivedcontent.

For example, in the case where the content-presentation device receivesa linear sequence of content segments that includes a givenadvertisement segment positioned somewhere within the sequence, it canbe desirable for the content-presentation device to replace the givenadvertisement segment with a different advertisement segment that isperhaps more targeted to the end-user (e.g., more targeted to theend-user's interests, demographics, etc.). As another example, it can bedesirable for the content-presentation device to overlay on the givenadvertisement segment, overlay content that enhances the givenadvertisement segment in a way that is again perhaps more targeted tothe end-user. The described content-modification system can facilitateproviding these and other related features.

Content modification may involve identifying a segment of content asbeing modifiable, and modifying the identified content either byreplacing with supplemental content, or by overlaying supplementalcontent on it. In either case, a modification operation may entailpresenting the supplemental content during a time interval originallyscheduled for presentation of the single segment of content by acontent-presentation device. In practice, there can be circumstances inwhich two or more segments of supplemental content may be identified forconsecutive, back-to-back content-modification operations by a givencontent-presentation device. Thus, instead of modification of a singlesegment of content, two or more consecutive segments of content may besubject to two or more consecutive content-modification operations by acontent-presentation device. In order to accommodate such situationswith efficiency and in a streamlined fashion, further operations may beused to “stitch” together, or concatenate, two or more segments ofsupplemental content so as to ensure contiguous download and delivery ofmultiple supplemental content segments to the content-presentationdevice, and to facilitate consecutive content-modification operationswith the multiple supplemental content segments.

Content modification with content stitching may require timingcircumstance in which the entirety of second supplemental content isreceived by the content-presentation device in sufficient time to beavailable for completion of a second presentation modificationoperation. However, this circumstance may not necessarily be guaranteed.For example, transmission and/or network delays could cause download tothe content-presentation device of the second supplemental content totake too long to complete in time to be available for the completesecond content-modification operation. In particular, if the secondpresentation-modification operation is a full segment replacement, andthe entire second supplemental content is not received by end of thesegment being replaced, there can be a gap during which no content atall is available for playout (presentation). To address this scenario,the content-presentation device may carry out additional and/or modifiedoperations to unschedule a second content-modification operation if itdetermines that second supplemental content will not be fully receivedin time.

Accordingly, example embodiments described herein that both accommodatecontent stitching in content-modification operations, and unschedulingof content-modification operations to avoid potential deleteriouseffects of incomplete receipt of supplemental content prior tocompletion of content-modification operations.

II. Architecture

A. Content-Modification system

FIG. 1 is a simplified block diagram of an example content-modificationsystem 100. The content-modification system 100 can include variouscomponents, such as a content-distribution system 102, acontent-presentation device 104, a fingerprint-matching server 106, acontent-management system 108, a data-management system 110, and/or asupplemental-content delivery system 112.

The content-modification system 100 can also include one or moreconnection mechanisms that connect various components within thecontent-modification system 100. For example, the content-modificationsystem 100 can include the connection mechanisms represented by linesconnecting components of the content-modification system 100, as shownin FIG. 1.

In this disclosure, the term “connection mechanism” means a mechanismthat connects and facilitates communication between two or morecomponents, devices, systems, or other entities. A connection mechanismcan be or include a relatively simple mechanism, such as a cable orsystem bus, and/or a relatively complex mechanism, such as apacket-based communication network (e.g., the Internet). In someinstances, a connection mechanism can be or include a non-tangiblemedium, such as in the case where the connection is at least partiallywireless. In this disclosure, a connection can be a direct connection oran indirect connection, the latter being a connection that passesthrough and/or traverses one or more entities, such as a router,switcher, or other network device. Likewise, in this disclosure,communication (e.g., a transmission or receipt of data) can be a director indirect communication.

The content-modification system 100 and/or components thereof can takethe form of a computing system, an example of which is described below.

Notably, in practice, the content-modification system 100 is likely toinclude many instances of at least some of the described components. Forexample, the content-modification system 100 is likely to include manycontent-distribution systems and many content-presentation devices.

B. Computing System

FIG. 2 is a simplified block diagram of an example computing system 200.The computing system 200 can be configured to perform and/or can performone or more operations, such as the operations described in thisdisclosure. The computing system 200 can include various components,such as a processor 202, a data-storage unit 204, a communicationinterface 206, and/or a user interface 208.

The processor 202 can be or include a general-purpose processor (e.g., amicroprocessor) and/or a special-purpose processor (e.g., a digitalsignal processor). The processor 202 can execute program instructionsincluded in the data-storage unit 204 as described below.

The data-storage unit 204 can be or include one or more volatile,non-volatile, removable, and/or non-removable storage components, suchas magnetic, optical, and/or flash storage, and/or can be integrated inwhole or in part with the processor 202. Further, the data-storage unit204 can be or include a non-transitory computer-readable storage medium,having stored thereon program instructions (e.g., compiled ornon-compiled program logic and/or machine code) that, upon execution bythe processor 202, cause the computing system 200 and/or anothercomputing system to perform one or more operations, such as theoperations described in this disclosure. These program instructions candefine, and/or be part of, a discrete software application.

In some instances, the computing system 200 can execute programinstructions in response to receiving an input, such as an inputreceived via the communication interface 206 and/or the user interface208. The data-storage unit 204 can also store other data, such as any ofthe data described in this disclosure.

The communication interface 206 can allow the computing system 200 toconnect with and/or communicate with another entity according to one ormore protocols. Therefore, the computing system 200 can transmit datato, and/or receive data from, one or more other entities according toone or more protocols. In one example, the communication interface 206can be or include a wired interface, such as an Ethernet interface or aHigh-Definition Multimedia Interface (HDMI). In another example, thecommunication interface 206 can be or include a wireless interface, suchas a cellular or WI-FI interface.

The user interface 208 can allow for interaction between the computingsystem 200 and a user of the computing system 200. As such, the userinterface 208 can be or include an input component such as a keyboard, amouse, a remote controller, a microphone, and/or a touch-sensitivepanel. The user interface 208 can also be or include an output componentsuch as a display device (which, for example, can be combined with atouch-sensitive panel) and/or a sound speaker.

The computing system 200 can also include one or more connectionmechanisms that connect various components within the computing system200. For example, the computing system 200 can include the connectionmechanisms represented by lines that connect components of the computingsystem 200, as shown in FIG. 2.

The computing system 200 can include one or more of the above-describedcomponents and can be configured or arranged in various ways. Forexample, the computing system 200 can be configured as a server and/or aclient (or perhaps a cluster of servers and/or a cluster of clients)operating in one or more server-client type arrangements, for instance.

As noted above, the content-modification system 100 and/or componentsthereof can take the form of a computing system, such as the computingsystem 200. In some cases, some or all these entities can take the formof a more specific type of computing system. For instance, in the caseof the content-presentation device 104, it can take the form of adesktop computer, a laptop, a tablet, a mobile phone, a television set,a set-top box, a television set with an integrated set-top box, a mediadongle, or a television set with a media dongle connected to it, amongother possibilities.

III. Example Operations

The content-modification system 100 and/or components thereof can beconfigured to perform and/or can perform one or more operations.Examples of these operations and related features will now be described.

As noted above, in practice, the content-modification system 100 islikely to include many instances of at least some of the describedcomponents. Likewise, in practice, it is likely that at least some ofdescribed operations will be performed many times (perhaps on a routinebasis and/or in connection with additional instances of the describedcomponents).

A. Operations Related to the Content-Distribution System TransmittingContent and the Content-Presenting Device Receiving and OutputtingContent

For context, general operations and examples related to thecontent-distribution system 102 transmitting content and thecontent-presentation device 104 receiving and outputting content willnow be described.

To begin, the content-distribution system 102 can transmit content(e.g., that it received from a content provider) to one or more entitiessuch as the content-presentation device 104. Content can be or includeaudio content and/or video content, for example. In some examples,content can take the form of a linear sequence of content segments(e.g., program segments and advertisement segments) or a portionthereof. In the case of video content, a portion of the video contentmay be one or more frames, for example.

The content-distribution system 102 can transmit content on one or morechannels (sometimes referred to as stations or feeds). As such, thecontent-distribution system 102 can be associated with a single channelcontent distributor or a multi-channel content distributor such as amulti-channel video program distributor (MVPD).

The content-distribution system 102 and its means of transmission ofcontent on the channel to the content-presentation device 104 can takevarious forms. By way of example, the content-distribution system 102can be or include a cable-television head-end that is associated with acable-television provider and that transmits the content on the channelto the content-presentation device 104 through hybrid fiber/coaxialcable connections. As another example, the content-distribution system102 can be or include a satellite-television head-end that is associatedwith a satellite-television provider and that transmits the content onthe channel to the content-presentation device 104 through a satellitetransmission. As yet another example, the content-distribution system102 can be or include a television-broadcast station that is associatedwith a television-broadcast provider and that transmits the content onthe channel through a terrestrial over-the-air interface to thecontent-presentation device 104. In these and other examples, thecontent-distribution system 102 can transmit the content in the form ofan analog or digital broadcast stream representing the content.

The content-presentation device 104 can receive content from one or moreentities, such as the content-distribution system 102. In one example,the content-presentation device 104 can select (e.g., by tuning to) achannel from among multiple available channels, perhaps based on inputreceived via a user interface, such that the content-presentation device104 can receive content on the selected channel.

In some examples, the content-distribution system 102 can transmitcontent to the content-presentation device 104, which thecontent-presentation device 104 can receive, and therefore thetransmitted content and the received content can be the same. However,in other examples, they can be different, such as where thecontent-distribution system 102 transmits content to thecontent-presentation device 104, but the content-presentation device 104does not receive the content and instead receives different content froma different content-distribution system.

The content-presentation device 104 can also output content forpresentation. As noted above, the content-presentation device 104 cantake various forms. In one example, in the case where thecontent-presentation device 104 is a television set (perhaps with anintegrated set-top box and/or media dongle), outputting the content forpresentation can involve the television set outputting the content via auser interface (e.g., a display device and/or a sound speaker), suchthat it can be presented to an end-user. As another example, in the casewhere the content-presentation device 104 is a set-top box or a mediadongle, outputting the content for presentation can involve the set-topbox or the media dongle outputting the content via a communicationinterface (e.g., an HDMI interface), such that it can be received by atelevision set and in turn output by the television set for presentationto an end-user.

As such, in various scenarios, the content-distribution system 102 cantransmit content to the content-presentation device 104, which canreceive and output the content for presentation to an end-user. In somesituations, even though the content-presentation device 104 receivescontent from the content-distribution system 102, it can be desirablefor the content-presentation device 104 to perform acontent-modification operation so that the content-presentation device104 can output for presentation alternative content instead of at leasta portion of that received content.

For example, in the case where the content-presentation device 104receives a linear sequence of content segments that includes a givenadvertisement segment positioned somewhere within the sequence, it canbe desirable for the content-presentation device 104 to replace thegiven advertisement segment with a different advertisement segment thatis perhaps more targeted to the end-user (i.e., more targeted to theend-user's interests, demographics, etc.). As another example, it can bedesirable for the content-presentation device 104 to overlay on thegiven advertisement segment, overlay content that enhances the givenadvertisement segment in a way that is again perhaps more targeted tothe end-user. The described content-modification system 100 canfacilitate providing these and other related features.

As noted above, in one example, content can take the form of a linearsequence of content segments. As such, in one example, thecontent-distribution system 102 can transmit a linear sequence ofcontent segments. This is referred to herein as a “transmissionsequence.” Likewise, the content-presentation device 104 can receive alinear sequence of content segments. This is referred to herein as a“receipt sequence.” In line with the discussion above, the transmissionsequence and the receipt sequence can be the same or they can bedifferent.

FIG. 3 illustrates some examples of these concepts. In one example, thetransmission sequence is the TRANSMISSION SEQUENCE 302 shown in FIG. 3.As shown, the TRANSMISSION SEQUENCE 302 includes a PROGRAM SEGMENT A302-A, followed by an AD SEGMENT B 302-B, followed by AD SEGMENT C302-C.

Likewise, in one example, the receipt sequence is the RECEIPT SEQUENCE304 shown in FIG. 3. In this example, the content-distribution system102 transmits the TRANSMISSION SEQUENCE 302 to the content-presentationdevice 104, which the content-presentation device 104 receives as theRECEIPT SEQUENCE 304, and therefore the TRANSMISSION SEQUENCE 302 andthe RECEIPT SEQUENCE 304 are the same. As such, as shown, the RECEIPTSEQUENCE 304 also includes the PROGRAM SEGMENT A 304-A, followed by theAD SEGMENT B 304-B, followed by the AD SEGMENT C 302-C.

In FIG. 3, the transmission time of the TRANSMISSION SEQUENCE 302 andthe receipt time of the RECEIPT SEQUENCE 304 are shown by way of theirrelationship to a TIMELINE 350. Notably, the transmission time and thereceipt time are offset from each other due to a content-transmissiondelay, which is described in greater detail below.

As noted above, in some situations, even though the content-presentationdevice 104 receives content from the content-distribution system 102, itcan be desirable for the content-presentation device 104 to perform acontent-modification operation so that the content-presentation device104 can output for presentation alternative content instead of at leasta portion of that received content. For example, in the case where thecontent-presentation device 104 receives the receipt sequence, ratherthan outputting for presentation the receipt sequence, thecontent-presentation device 104 can output for presentation a modifiedversion of the receipt sequence instead. This is referred to herein as a“modified sequence.”

For example, in the case where the receipt sequence includes a givenadvertisement segment positioned somewhere within the receipt sequence,it can be desirable for the content-presentation device 104 to replacethe given advertisement segment with a different advertisement segmentthat is perhaps more targeted to the end-user (i.e., more targeted tothe end-user's interests, demographics, etc.), thereby resulting in amodified sequence that the content-presentation device 104 can outputfor presentation.

To illustrate this, in one example, the modified sequence is the FIRSTMODIFIED SEQUENCE 306 shown in FIG. 3. As shown, the FIRST MODIFIEDSEQUENCE 306 includes the PROGRAM SEGMENT A 306-A, followed by the ADSEGMENT D 306-D (which replaced the AD SEGMENT B 304-B), followed by ADSEGMENT C 306-C.

As another example, it can be desirable for the content-presentationdevice 104 to overlay, on the given advertisement segment, overlaycontent that enhances the given advertisement segment in a way that isagain perhaps more targeted to the end-user, thereby resulting in amodified sequence that the content-presentation device 104 can outputfor presentation.

To illustrate this, in another example, the modified sequence is theSECOND MODIFIED SEQUENCE 308 shown in FIG. 3. As shown, the SECONDMODIFIED SEQUENCE 308 includes the PROGRAM SEGMENT A 308-A, followed bythe AD SEGMENT B′ 308-B′ (which is the AD SEGMENT B 304-B modified withoverlay content), followed by AD SEGMENT C 308-C.

Other portions of FIG. 3 will be described later in this disclosure asrelated concepts are introduced and described.

Moving on in view of the context provided above, FIGS. 4A, 4B, 4C, 4D,4E and 4F, collectively make up a table showing example time-periods andcorresponding operations that can be performed in connection with thecontent-modification system 100. These and other related operations willnow be described.

B. Operations Related to the Content-Distribution System TransmittingFirst Content on a Channel

During a time-period T1, the content-distribution system 102 cantransmit content on a channel to the content-presentation device 104.This content is referred to herein as “first content.” In one example,the first content is the FIRST CONTENT 310 shown in FIG. 3.

During a time-period T2, the content-distribution system 102 cangenerate fingerprint data representing the first content. Thisfingerprint data is referred to herein as “first fingerprint data.” Thecontent-distribution system 102 can generate the first fingerprint datausing any content fingerprinting process now known or later developed.An example fingerprint generation technique is described in U.S. Pat.No. 9,495,451 issued Nov. 15, 2016, the entirety of which is herebyincorporated by reference herein. The content-distribution system 102can generate first fingerprint data at a given rate, such as at the rateof one fingerprint per frame of the first content. The first fingerprintdata can be or include some or all of these generated fingerprints.

The content-distribution system 102 can access the first content atvarious points within the content-distribution system 102. As oneexample, the content-distribution system 102 can access the firstcontent after it is output by a distribution amplifier within thecontent-distribution system 102.

Also during the time-period T2, the content-distribution system 102 cangenerate metadata associated with the first content and/or the firstfingerprint data. This metadata is referred to herein as “firstmetadata.” In one example, the first metadata can be or include atransmission time-stamp, which represents a time-point at which thecontent-distribution system 102 transmitted the first content. Thecontent-distribution system 102 can determine the transmissiontime-stamp in various ways, such as based on a time clock that issynchronized to a reference clock.

As another example, the first metadata can be or include a channelidentifier, which identifies the channel on which thecontent-distribution system 102 is transmitting the first content. Thecontent-distribution system 102 can determine the channel identifier invarious ways such as based on mapping data that maps thecontent-distribution system 102 and/or physical inputs and/or outputswithin the content-distribution system 102 to respective channelidentifiers. In one example, in the case where the content-distributionsystem 102 transmits content A on channel A, content B on channel B, andcontent C on channel C, the mapping data can specify which of threedifferent outputs (perhaps on three different distribution amplifiers)maps to which channel identifier, such that the content-distributionsystem 102 can determine the appropriate channel identifier for contentof a given channel.

As another example, the first metadata can be or include SCTE-104 data,a watermark, or a similar type of metadata, any of which can themselvesencode other metadata, such as a program identifier, an advertisementidentifier (e.g., an industry standard coding identification (ISCI)key), a program genre, or another type of textual or numeric metadata,for instance.

The content-distribution system 102 can associate the first fingerprintdata with the first metadata in various ways. For instance, in the casewhere the first fingerprint data includes multiple fingerprints witheach fingerprint representing a corresponding frame of the firstcontent, the content-distribution system 102 can associate eachfingerprint with a corresponding transmission time-stamp and/or withother corresponding first metadata.

During a time-period T3, the content-distribution system 102 cantransmit the first fingerprint data and the first metadata to thefingerprint-matching server 106. The content-distribution system 102 cantransmit the first fingerprint data and the first metadata at a giveninterval. For example, every two seconds, the content-distributionsystem 102 can transmit the first fingerprint data and the firstmetadata that it generated during that most recent two-secondtime-period.

C. Operations Related to the Content-Presentation Device ReceivingSecond Content

During a time-period T4, the content-presentation device 104 can receivecontent from the content-distribution system 102. This content isreferred to herein as “second content.” In one example, the secondcontent is the SECOND CONTENT 312 shown in FIG. 3.

During a time-period T5, the content-presentation device 104 cangenerate fingerprint data representing the second content. Thisfingerprint data is referred to herein as “second fingerprint data.” Thecontent-presentation device 104 can generate the second fingerprint datausing any content fingerprinting process now known or later developed.The content-presentation device 104 can generate the second fingerprintdata at various rates, such as at the rate of one fingerprint per frameof the second content. The second fingerprint data can be or includesome or all of these generated fingerprints.

The content-presentation device 104 can access the second content atvarious points within the content-presentation device 104. As oneexample, the content-presentation device 104 can access the secondcontent as it is being received by an input buffer (e.g., an HDMIbuffer) of the content-presentation device 104. In anotherconfiguration, the content-presentation device 104 can access the secondcontent as it is being received by a display buffer of thecontent-presentation device 104. In this configuration, the secondcontent can therefore be content that the content-presentation device104 not only receives, but also outputs for presentation.

Also during the time-period T5, the content-presentation device 104 cangenerate metadata associated with the second content and/or the secondfingerprint data. This metadata is referred to herein as “secondmetadata.” As one example, the second metadata can be or include areceipt time-stamp, which represents a time-point at which thecontent-presentation device 104 received the second content. Thecontent-presentation device 104 can determine the receipt time-stamp invarious ways, such as based on a time clock that is synchronized to areference clock. As noted above, the content-presentation device 104 canaccess the second content at various points within thecontent-presentation device 104. In one example, the point at which thesecond content is accessed can be considered the “receipt” point forpurposes of determining the receipt time-stamp.

In practice, while the first metadata is likely to be or include achannel identifier, the second metadata is likely to not be nor includea channel identifier.

The content-presentation device 104 can associate the second fingerprintdata with the second metadata in various ways. For instance, where thesecond fingerprint data includes multiple fingerprints with eachfingerprint representing a corresponding frame of second content, thecontent-presentation device 104 can associate each second fingerprintwith a corresponding receipt time-stamp and/or other correspondingmetadata.

During a time-period T6, the content-presentation device 104 cantransmit the second fingerprint data and the second metadata to thefingerprint-matching server 106. The content-presentation device 104 cantransmit the second fingerprint data and the second metadata at a giveninterval. For example, every two seconds, the content-presentationdevice 104 can transmit the second fingerprint data and the secondmetadata that it generated during that most recent two-secondtime-period.

D. Operations Related to Identifying a Channel on Which theContent-Presentation Device is Receiving the Second Content

During a time-period T7, the fingerprint-matching server 106 can receivethe first fingerprint data and the first metadata from thecontent-distribution system 102. As noted above, the first fingerprintdata represents the first content transmitted by thecontent-distribution system 102 on the channel. As noted above, thefirst metadata can, and for the purposes of this described example does,identify the channel. In this way, the first content can be consideredas content being transmitted on an identified channel.

During a time-period T8, the fingerprint-matching server 106 can receivethe second fingerprint data and the second metadata from thecontent-presentation device 104. As noted above, the second fingerprintdata represents the second content received by the content-presentationdevice 104. However, as noted above, the associated metadata may not,and for the purposes of this described example does not, identify thechannel. In this way, the second content can be considered as contentbeing received on an unidentified channel.

During a time-period T9, the fingerprint-matching server 106 can comparethe first fingerprint data and the second fingerprint data to determinewhether there is a match. In this disclosure, this type of matchattempt, namely a match attempt between (i) reference fingerprint datarepresenting content being transmitted on an identified channel and (ii)query fingerprint data representing content being received on anunidentified channel, is referred to herein as a “cold match attempt.”

During a time-period T10, based on the comparing, thefingerprint-matching server 106 can detect a match between the firstfingerprint data and the second fingerprint data. Thefingerprint-matching server 106 can compare and/or detect a matchbetween fingerprint data using any content fingerprint comparing andmatching technique now known or later developed. An example fingerprintcomparing and matching technique is described in U.S. Pat. No. 9,146,990issued Sep. 29, 2015, the entirety of which is hereby incorporated byreference herein.

To effectively compare the first fingerprint data and the secondfingerprint data, the fingerprint-matching server 106 may need toaccount for a content-transmission delay. For context, in the case wherethe content-distribution system 102 transmits a given frame of contenton a given channel at a time-point A, for various reasons, thecontent-presentation device 104 may not receive that frame until atime-point B that is later (e.g., ten seconds later) than the time-pointA. This type of delay is referred to herein as a “content-transmissiondelay.”

In one example, the time-point A, the time-point B, and thecontent-transmission delay can be the TIME-POINT A 314, the TIME-POINT B316, and the CONTENT-TRANSMISSION DELAY 318, respectively, shown FIG. 3.Note that FIG. 3 is for illustration purposes and is not necessarily toscale at least with respect to time. In practice, the actual amount ofcontent-transmission delay may be different from the amount shown.

To help the fingerprint-matching server 106 effectively compare thefirst fingerprint data with the second fingerprint data, thefingerprint-matching server 106 may need to account for such acontent-transmission delay. In one example, the fingerprint-matchingserver 106 can do this by comparing the first fingerprint data that itreceives at a receipt time-point with the second fingerprint data thatit receives during a time-period defined by a starting time-point and anending time-point. The starting time-point can be the receipt time-pointplus an offset representing an anticipated content-transmission delay(e.g., ten seconds), minus a tolerance a time-period (e.g., twoseconds). The ending time-point can be the receipt time-point plus theoffset (e.g., ten seconds), plus the tolerance a time-period (e.g., twoseconds). As such, in one example where the anticipatedcontent-transmission delay is 10 seconds, the fingerprint-matchingserver 106 can compare first fingerprint data that it receives at areceipt time-point with second fingerprint data that it receives duringa time-period between (i) the receipt time-point plus eight seconds and(ii) receipt time-point plus twelve seconds.

In some cases, the fingerprint-matching server 106 can determine acontent-transmission delay, which it can use to select an appropriateoffset for use in determining the starting and ending time-points, asdescribed above. The fingerprint-matching server 106 can determine thecontent-transmission delay in various ways. For example, after thefingerprint-matching server 106 detects a match based on a cold matchattempt, the fingerprint-matching server 106 can determine thecontent-transmission delay as a difference between the correspondingtransmission time-stamp (of the first metadata) and the correspondingreceipt time-stamp (of the second metadata), for example. Notably, thecontent-transmission delay can vary from channel to channel.

During a time-period T11, based on the detected match, thefingerprint-matching server 106 can identify the channel on which thesecond content is being received by the content-presentation device 104.In one example, the fingerprint-matching server 106 can identify thechannel based on the channel identifier metadata associated with thefirst fingerprint data used to detect the match.

Notably, in practice, since there are likely to be multiple potentialchannels on which the content-presentation device 104 is receiving thesecond content, the fingerprint-matching server 106 is likely to comparethe second fingerprint data with multiple instances of first fingerprintdata (each representing a different respective instance of first contenton a different respective channel), to determine which of those multipleinstances matches the second fingerprint data.

Also, in some cases, the fingerprint-matching server 106 can detect amatch between the second fingerprint data and each of multiple instancesof first fingerprint data (each representing a different respectiveinstance of first content on a different respective channel). This isreferred to herein as a “multimatch scenario” and can occur for variousreasons. For example, this can occur when the content-distributionsystem 102 is transmitting the same or similar content on more than onechannel at or about the same time. In this scenario, thefingerprint-matching server 106 can perform additional operations toidentity, from among the multiple channels associated with themultimatch scenario, on which specific channel the content-presentationdevice 104 is receiving the second content. The fingerprint-matchingserver 106 can do this using any channel multimatch disambiguationtechnique now known or later developed. An example channel multimatchdisambiguation technique is described in U.S. Pat. No. 9,992,533 issuedJun. 5, 2018, the entirety of which is hereby incorporated by referenceherein.

E. Operations Related to Determining Historical Content Consumption Data

During a time-period T12, the fingerprint-matching server 106 cangenerate metadata associated with the identified channel. For example,the metadata can be or include a channel identification time-stamp. Thefingerprint-matching server 106 can determine the channel identificationtime-stamp in various ways, such as based on a time clock that issynchronized to a reference clock. In another example, the metadata canbe or include a device identifier that identifies thecontent-presentation device 104 that is receiving content on theidentified channel. The fingerprint-matching server 106 can determinethe device identifier in various ways, such as by receiving it from thecontent-presentation device 104. In another example, thefingerprint-matching server 106 can receive data (e.g., deviceregistration data) from the content-presentation device 104 and can usemapping data to map the received data to determine the deviceidentifier.

During a time-period T13, the fingerprint-matching server 106 cantransmit an indication of the identified channel and the associatedmetadata to the data-management system 110.

During a time-period T14, the data-management system 110 can receive theindication of the identified channel and the associated metadata fromthe fingerprint-matching server 106.

The data-management system 110 can use the received indication of theidentified channel and the associated metadata, perhaps with other data,to determine when the content-presentation device 104 has receivedcontent on the identified channel, what specific content thecontent-presentation device 104 has received, etc. This type of data isreferred to herein as “historical content consumption data.”

As such, during a time-period T15, the data-management system 110 candetermine historical content consumption data associated with thecontent-presentation device 104.

F. Operations Related to the Content-Distribution System TransmittingThird Content

As noted above, the fingerprint-matching server 106 can identify thechannel on which the content-presentation device 104 is receiving thesecond content.

During a time-period T16, the content-distribution system 102 cantransmit content on the identified channel to the content-presentationdevice 104. This content is referred to herein as “third content.” Inone example, the third content is the THIRD CONTENT 320 shown in FIG. 3.In practice, the content-distribution system 102 is likely to transmitthe third content shortly after (e.g., immediately after or a fewseconds or minutes after) transmitting the first content.

During a time-period T17, the content-distribution system 102 cangenerate fingerprint data representing the third content. Thisfingerprint data is referred to herein as “third fingerprint data.”

Also during the time-period T17, the content-distribution system 102 cangenerate metadata associated with the third content and/or the thirdfingerprint data. This metadata is referred to herein as “thirdmetadata.” The content-distribution system 102 can also associate thethird fingerprint data with the third metadata.

During a time-period T18, the content-distribution system 102 cantransmit the third fingerprint data and the third metadata to thefingerprint-matching server 106.

The content-distribution system 102 can transmit the third content,generate the third fingerprint data, generate the third metadata,associate the third fingerprint data with the third metadata, andtransmit the third fingerprint data and the third metadata in variousways, such as ways that are the same as or similar to those describedabove in connection with transmitting the first content, generating thefirst fingerprint data, generating the first metadata, associating thefirst fingerprint data with the first metadata, and transmitting thefirst fingerprint data and the first metadata.

G. Operations Related to the Content-Management System Receiving aModifiable Content-Segment

During a time-period T19, the content-management system 108 can receivecontent in the form of a content segment that has been identified as acandidate to be modified. This content is referred to herein as a“modifiable content-segment” or “fourth content.” In one example, themodifiable content-segment is the MODIFIABLE CONTENT-SEGMENT 322 shownin FIG. 3.

The modifiable content-segment can take various forms. For example, themodifiable content-segment can be an advertisement segment (e.g., acommercial) or a program segment. As such, in one example, themodifiable content-segment can be an advertisement segment that has beenidentified as a candidate to be modified, perhaps by way of beingreplaced with a different advertisement segment, and/or by way of havingcontent overlaid thereon.

In one example, a user, perhaps associated with the content-distributionsystem 102, can facilitate uploading the modifiable content-segment tothe content-management system 108, such that the content-managementsystem 108 can receive it in this way.

During a time-period T20, the content-management system 108 can generatefingerprint data representing the modifiable content-segment. Thisfingerprint data is referred to herein as “fourth fingerprint data.” Thecontent-management system 108 can generate the fourth fingerprint datausing any fingerprint generation technique now known or later developed.The content-management system 108 can generate the fourth fingerprintdata at a given rate, such as at the rate of one fingerprint per frameof the fourth content. The fourth fingerprint data can be or includesome or all of these generated fingerprints.

Also during the time-period T20, the content-management system 108 cangenerate metadata associated with the modifiable content-segment and/orthe fourth fingerprint data. This metadata is referred to herein as“fourth metadata.” As one example, the fourth metadata can be or includea duration of the modifiable content-segment. The content-managementsystem 108 can determine the duration in various ways, such as based onthe fingerprint generation process. For example, in the case where thecontent-management system 108 generating the fourth fingerprint datainvolves generating one fingerprint per frame, where the modifiablecontent-segment has a frame rate of 30 frames per second, and where thefingerprinting process results in 300 fingerprints being generated, thecontent-management system 108 can deduce that the modifiablecontent-segment has a duration of ten seconds. The metadata can also beor include other information about the modifiable content-segment, suchas a content segment identifier, a title, and/or specifics aboutpermissible ways in which the modifiable content-segment can bemodified, etc.

During a time-period T21, the content-management system 108 can transmitthe fourth fingerprint data and the fourth metadata to thefingerprint-matching server 106.

In practice, the content-management system 108 is likely to receive manymodifiable content-segments. In such situations, the content-managementsystem 108 can perform one or more of the operations described above, asappropriate for each of the many received modifiable content-segments.As such, the content-management system 108 can transmit many instancesof fourth fingerprint data, each corresponding with a differentrespective modifiable content-segment, to the fingerprint-matchingserver 106.

H. Operations Related to the Fingerprint-Matching Server Identifying anUpcoming Content-Modification Opportunity on the Identified Channel

During a time-period T22, the fingerprint-matching server 106 canreceive the third fingerprint data and the third metadata from thecontent-distribution system 102. As noted above, the third fingerprintdata represents the third content transmitted by thecontent-distribution system 102 on the identified channel.

During a time-period T23, the fingerprint-matching server 106 canreceive the fourth fingerprint data and the fourth metadata from thecontent-management system 108. As noted above, the fourth fingerprintdata represents the modifiable content-segment.

During a time-period T24, the fingerprint-matching server 106 cancompare at least a portion of the third fingerprint data with at least aportion of the fourth fingerprint data to determine whether there is amatch.

During a time-period T25, based on the comparing, thefingerprint-matching server 106 can detect a match between the at leasta portion of the third fingerprint data and the at least a portion ofthe fourth fingerprint data. The fingerprint-matching server 106 cancompare and/or detect a match between fingerprint data using any contentfingerprint comparing and matching process now known or later developed.

During a time-period T26, based on the detected match, thefingerprint-matching server 106 can determine that at least a portion ofthe modifiable content-segment is included within the third content, andtherefore can identify an upcoming content-modification opportunity onthe identified channel. For example, the fingerprint-matching server 106can determine that at least a beginning portion of the MODIFIABLECONTENT-SEGMENT 322 is included within the THIRD CONTENT 320, as shownin FIG. 3, and therefore can identify an upcoming content-modificationopportunity.

As noted above, the fingerprint-matching server 106 can generate thirdmetadata, which can be the same as or similar to the first metadata. Assuch, the third metadata can be or include a transmission time-stampand/or a channel identifier, for example. However, the third metadatacan also be or include a position of at least a portion of themodifiable content-segment within the third content. In one example, themetadata can specify this using a starting frame marker and an endingframe market, each corresponding with a respective frame of the thirdcontent. The fingerprint-matching server 106 can determine the startingframe marker and the ending frame marker based on the matching.

Notably, in practice, since there are likely to be multiple potentialmodifiable content-segments where portions thereof could be includedwithin the third content, the fingerprint-matching server 106 is likelyto compare at least a portion of the third fingerprint data with atleast a portion of multiple instances of fourth fingerprint data (eachrepresenting a different respective instance of a modifiablecontent-segment), to determine which of those multiple instances of thefourth fingerprint data has a portion that matches the at least aportion of the third fingerprint data.

I. Operations Related to Preparing the Content-Presentation Device toPerform a Content-Modification Operation in Connection with theIdentified Upcoming Content-Modification Opportunity

During a time-period T27, based on the detected match, thefingerprint-matching server 106 can transmit the third fingerprint dataand the third metadata to the content-presentation device 104 data tofacilitate preparing the content-presentation device 104 to perform acontent-modification operation in connection with the identifiedupcoming content-modification opportunity.

During a time-period T28, the content-presentation device 104 canreceive the third fingerprint data and the third metadata from thefingerprint-matching server 106.

During a time-period T29, the content-presentation device 104 canreceive content on the identified channel. This content is referred toherein as “fifth content.” In one example, the fifth content is theFIFTH CONTENT 324 shown in FIG. 3.

For various reasons (e.g., due to a transmission delay associated withtransmitting fingerprint data and metadata being shorter that thecontent-transmission delay), the content-presentation device 104 canreceive the third fingerprint data and the third metadata from thefingerprint-matching server 106 before receiving the fifth content fromthe content-distribution system 102. In this way, thecontent-presentation device 104 can receive fingerprint datarepresenting content that the content-presentation device 104 isexpecting to receive shortly thereafter, and that thecontent-presentation device should actually receive shortly thereafterunless an interruption event (e.g., a channel-change event) occurs.

In practice, similar to how the content-distribution system 102 islikely to transmit the third content shortly after (e.g., immediatelyafter or a few seconds or minutes after) transmitting the first content,the content-presentation device 104 is likely to receive the fifthcontent shortly after (e.g., immediately after or a few seconds orminutes after) receiving the second content.

During a time-period T30, the content-presentation device 104 can outputfor presentation at least a portion of the fifth content. For example,referring to FIG. 3, the content-presentation device can output forpresentation the portion of the FIFTH CONTENT 324 that is the endportion of the PROGRAM SEGMENT A 304-A.

As noted above, in some situations, even though the content-presentationdevice 104 receives content from the content-distribution system 102, itcan be desirable for the content-presentation device 104 to perform acontent-modification operation so that the content-presentation device104 can output for presentation alternative content instead of at leasta portion of the received content.

As such, even though the content-presentation device 104 receives thefifth content and outputs for presentation at least a portion of thefifth content, it can be desirable for the content-presentation device104 to perform a content-modification operation so that thecontent-presentation device 104 can also output for presentationalternative content instead of at least another portion (e.g., theremaining portion) of the fifth content. For example, referring to FIG.3, it can be desirable for the content-presentation device 104 toreplace at least a portion of the AD SEGMENT B 304-B with at least aportion of a different advertisement segment that is perhaps moretargeted to the end-user. As another example, it can be desirable forthe content-presentation device 104 to overlay on at least a portion ofthe AD SEGMENT B 304-B, overlay content that enhances at least a portionof the AD SEGMENT B 304-B in a way that is again perhaps more targetedto the end-user.

During a time-period T31, the content-presentation device 104 cangenerate fingerprint data representing the fifth content. Thisfingerprint data is referred to herein as “fifth fingerprint data.” Thecontent-distribution system 102 can generate the fifth fingerprint datausing any content fingerprinting process now known or later developed.The content-presentation device 104 can generate the fifth fingerprintdata at various rates, such as at the rate of one fingerprint per frameof the fifth content. The fifth fingerprint data can be or include someor all of these generated fingerprints.

Also during the time-period T31, the content-presentation device 104 cangenerate metadata associated with the fifth content and/or the fifthfingerprint data. This metadata is referred to herein as “fifthmetadata.”

The content-presentation device 104 can receive the fifth content,generate the fifth fingerprint data, generate the fifth metadata,associate the fifth fingerprint data with the fifth metadata in variousways, such as ways that are the same as or similar to those describedabove in connection with receiving the second content, generating thesecond fingerprint data, generating the second metadata, and associatingthe second fingerprint data with the second metadata.

As noted above, the content-presentation device 104 can receive thethird fingerprint data from the fingerprint-matching server 106 and cangenerate the fifth fingerprint data.

During a time-period T32, the content-presentation device 104 cancompare the third fingerprint data and the fifth fingerprint data todetermine whether there is a match.

During a time-period T33, based on the comparing, thecontent-presentation device 104 can detect a match between the thirdfingerprint data and the fifth fingerprint data. In this disclosure,this type of match attempt, namely a match attempt between (i) referencefingerprint data representing content transmitted by thecontent-distribution system 102 on an identified channel (at least basedon the most recent channel identification analysis), and (ii) queryfingerprint data representing content being received by thecontent-presentation device 104 on the same identified channel, isreferred to herein as a “hot match attempt.” The fingerprint-matchingserver 106 can compare and/or detect a match between fingerprint datausing any content fingerprint comparing and matching process now knownor later developed.

During a time-period T34, based on the detected match, thecontent-presentation device 104 can determine a time-point at which theidentified upcoming modification opportunity starts. This is referred toherein as the “modification start-time.” In one example, themodification start-time is the MODIFICATION START-TIME 326 as shown FIG.3.

In one example, the content-presentation device 104 can determine themodification start-time by starting with the transmission time-stampassociated with the starting frame marker (which, as described above,can be or be included in the third metadata) and adding thecontent-transmission delay to that transmission time-stamp, to arrive atthe modification start-time.

As another example, the content-presentation device 104 can determinethe modification start-time by first establishing a synchronous lockbetween the third content, the third fingerprint data, and/or the thirdmetadata on the one hand, and the fifth content, the third fifthfingerprint data, and/or the fifth metadata, on the other hand. Thecontent-presentation device 104 can establish the synchronous lock usingany synchronous lock technique now known or later developed. An examplesynchronous lock technique is described in U.S. Pat. No. 10,506,275issued Dec. 10, 2019, the entirety of which is hereby incorporated byreference herein. Another example synchronous lock technique isdescribed in U.S. Pat. App. Pub. No. 2020/0029108 published Jan. 23,2020, the entirety of which is hereby incorporated by reference herein.

The content-presentation device 104 can then determine the modificationstart-time by determining a time-period between (i) a current receipttime-stamp associated with a first portion of the fifth content that thecontent-presentation device 104 is currently receiving and (ii) based onthe synchronous lock, an anticipated receipt time-stamp associated witha second portion of the fifth content that is the start of themodifiable content-segment, and then adding the determined time-periodto the current receipt time-stamp, to arrive at the modificationstart-time.

Also during the time-period T34, based on the detected match, thecontent-presentation device 104 can determine a time-point at which theidentified upcoming modification opportunity ends. This is referred toherein as the “modification end-time.” In one example, the modificationend-time is the MODIFICATION END-TIME 328 as shown FIG. 3.

In one example, the content-presentation device 104 can determine themodification end-time by starting with the modification start-time andadding the duration of the modifiable content-segment (which, asdescribed above, can be or be included in the fourth metadata) to themodification start-time, to arrive at the modification end-time.

Notably, if the content-presentation device 104 performs a hot matchattempt and does not detect a match, the content-presentation device 104can determine that the content-presentation device 104 is no longerreceiving content on the most recently identified channel. In response,the content-presentation device 104 can repeat one or more of theoperations described above so that the fingerprint-matching server 106can perform another cold match attempt, to attempt to identify thechannel again.

During a time-period T35, the content-presentation device 104 cantransmit a request for content for use in connection with performing thecontent-modification operation, to the content-management system 108.This content is referred to herein as “supplemental content.” In oneexample, the content-presentation device 104 can transmit the requestbefore the modification start-time (e.g., ten seconds before). In somecases, the request can include selection criteria for the supplementalcontent, which the content-presentation device 104 can determine basedon the third metadata that the content-presentation device 104 receivesfrom the fingerprint-matching server 106, for instance.

For example, the selection criteria can specify a requested type ofcontent (e.g., a replacement content segment or overlay content),duration (e.g., 15 seconds, 30 seconds, or 60 seconds), aspect ratio(e.g., 4:3 or 16:9), and/or resolution (e.g., 720p or 1080p).

During a time-period T36, the content-management system 108 can receivethe request and use it as a basis to select supplemental content fromamong multiple supplemental content items that are available forselection. In some cases, the content-management system 108 can receiveand consider various data to help inform which supplemental content toselect. For example, the content-management system 108 can receivehistorical content consumption data for the content-presentation device104 from the data-management system 110 and/or the content-managementsystem 108 can receive demographic data from a demographic dataprovider. The content-management system 108 can then use at least thereceived historical content consumption data and/or the receiveddemographic data as a basis to select the supplemental content.

The content-management system 108 can cause the selected supplementalcontent to be transmitted to the content-presentation device 104. In oneexample, the content-management system 108 can do this by communicatingwith a supplemental-content delivery system 112 that can host thesupplemental content. The supplemental-content delivery system 112 cantake various forms and can include various components, such as a contentdistribution network (CDN).

During a time-period T37, the content-management system 108 can transmita request for a link (e.g., a Uniform Resource Identifier (URI) or aUniform Resource Locator (URL)) pointing to the hosted supplementalcontent, to the supplemental-content delivery system 112.

During a time-period T38, the supplemental-content delivery system 112can receive and respond to the request for the link by transmitting therequested link to the content-management system 108.

During a time-period T39, the content-management system 108 can then inturn transmit the link to the content-presentation device 104.

During a time-period T40, the content-presentation device 104 canreceive the link, which it can use to retrieve the supplemental contentfrom the supplemental-content delivery system 112, such that thecontent-presentation device 104 can use the retrieved supplementalcontent in connection with performing the content-modificationoperation. In one example, the content-presentation device 104 canretrieve the supplemental content and store the supplemental content ina data-storage unit of the content-presentation device 104.

As such, in some examples, the content-presentation device 104 canreceive the modifiable content-segment from one source (e.g., thecontent-distribution system 102), and the supplemental content fromanother source (e.g., the supplemental-content delivery system 112).These segments can be transmitted to, and received by, thecontent-presentation device 104 in different ways. For example, thecontent-distribution system 102 can transmit, and thecontent-presentation device 104 can receive, the modifiablecontent-segment as a broadcast stream transmission, whereas thesupplemental-content delivery system 112 can transmit, and thecontent-presentation device 104 can receive, the supplemental content asan over-the-top (OTT) transmission. In this context, in one example, thecontent-distribution system 102 can receive the modifiablecontent-segment via one communication interface (e.g., an HDMIinterface), and the content-presentation device 104 can receive thesupplemental content via a different communication interface (e.g., anEthernet or WI-FI interface).

J. Operations Related to the Content-Presentation Device Performing aContent-Modification Operation

At a time-period T41, the content-presentation device 104 can performthe content-modification operation. The content-presentation device 104can do this in various ways, perhaps depending on the type ofcontent-modification operation to be performed.

In one example, the content-presentation device 104 performing acontent-modification operation can involve the content-presentationdevice 104 modifying the modifiable content-segment by replacing it withsupplemental content. This is referred to herein as a“content-replacement operation.” For example, in this scenario, thecontent-presentation device 104 can receive a linear sequence of contentsegments that includes the modifiable content-segment and the associatedmetadata, and can also receive the supplemental content segment, asdescribed above. The content-presentation device 104 can output forpresentation the sequence of content segments up until the modificationstart-time (which corresponds to the start of the modifiablecontent-segment), at which time the content-presentation device 104 canswitch to outputting for presentation the supplemental content instead.Then, at the modification end-time (which corresponds to the end of themodifiable content-segment), the content-presentation device 104 canswitch back to outputting for presentation the content that follows inthe linear sequence of content segments (or perhaps to other content,such as additional supplemental content that is replacing anothermodifiable content-segment).

In one example, the operation of the content-presentation device 104switching from outputting the sequence of content segments to outputtingthe supplemental content can involve using various buffers of thecontent-presentation device 104. For example, this can involve thecontent-presentation device 104 switching from using first data in afirst input buffer where the sequence of content segments is beingreceived to using second data in a second input buffer where thesupplemental content is being received, to populate a display buffer.

As such, according to one example as illustrated in FIG. 3, byperforming a content replacement operation, the content-presentationdevice 104 can replace the AD SEGMENT B 302-B with the AD SEGMENT B304-B. As a result, rather than outputting for presentation the RECEIPTSEQUENCE 304, the content-presentation device can instead output forpresentation the FIRST MODIFIED SEQUENCE 306.

In another example, the content-presentation device 104 performing acontent-modification operation can involve the content-presentationdevice 104 modifying a modifiable content-segment by overlaying on themodifiable content-segment, overlay content (referred to herein as a“content overlay operation”). For example, in this scenario, thecontent-presentation device 104 can again receive a linear sequence ofcontent segments that includes the modifiable content-segment and theassociated metadata, and the content-presentation device 104 can alsoreceive the supplemental content, as described above.

The content-presentation device 104 can then output for presentation themodifiable content-segment as it ordinarily would, except that startingat the modification start-time, the content-presentation device 104 canstart overlaying the supplemental content on the modifiablecontent-segment. The content-presentation device 104 can continueoverlaying the supplemental content until the modification end-time. Inthis way, the content-presentation device 104 can overlay thesupplemental content during at least some temporal portion of themodifiable content-segment.

In one example, the operation of the content-presentation device 104overlaying supplemental content on the modifiable content-segment caninvolve using various buffers of the content-presentation device 104.For example, this can involve the content-presentation device 104 usinga portion of first data in a first input buffer where the sequence ofcontent segments is being received together with second data in a secondinput buffer where the supplemental content is being received, for thepurposes of populating a display buffer. In this way, thecontent-presentation device can combine relevant portions of themodifiable content-segment (i.e., all portions except those representingregion where the supplemental content is to be overlaid) together withthe supplemental content to be used as an overlay, to create the desiredmodifiable content-segment plus the supplemental content overlaidthereon.

As such, according to one example as illustrated in FIG. 3, byperforming a content overlay operation, the content-presentation device104 can overlay supplemental content on the AD SEGMENT B 302-B, therebymodifying it to AD SEGMENT B 304-B′. As a result, rather than outputtingfor presentation the RECEIPT SEQUENCE 304, the content-presentationdevice can instead output for presentation the SECOND MODIFIED SEQUENCE308.

K. Tracking and Reporting Operation-Related Data

To help facilitate performance of various operations such as thecontent-presentation device 104 performing a content-modificationoperation and to help allow for the tracking and reporting of suchoperations, the content-modification system 100 and/or componentsthereof can track and report various operation-related data at varioustimes and in various ways.

As just a few illustrative examples, responsive to certain operationsbeing performed, such as those described herein, thefingerprint-matching server 106, the content-presentation device 104,and/or another entity can generate, store, and/or transmit messages thatindicate (i) that a modifiable content-segment has been identified, (ii)that a channel has been identified/confirmed (perhaps based on a matchdetected as a result of a cold or hot match attempt), (iii) that anupcoming content-modification opportunity on the identified channel hasbeen identified, (iv) that supplemental content has been requested, (v)that supplemental content has been received, (vi), that acontent-modification operation has started, (vii) that acontent-modification operation has ended, and/or (viii) that a scheduledcontent-modification operation was aborted and/or not performed for anygiven reason. In some cases, these messages can include other metadatarelated to these operations. For example, the metadata can specifyrelevant timing information, device identifiers, channel identifiers,content segment identifiers, etc.

L. Watermark-Based Techniques

Although this disclosure has described the content-modification system100 using fingerprint-based technology to perform various operations andto provide various features, in some examples, the content-modificationsystem 100 can use watermark-based techniques instead of, or in additionto, fingerprint-based techniques, to perform these and other operationsand to provide these and other features.

For example, as an alternative to the fingerprint-based techniquedescribed above in which the fingerprint-matching server 106 identifiesthe channel on which the second content is being received by thecontent-presentation device 104, the content-distribution system 102 oranother entity can insert a channel identifier in the form of awatermark into the second content, such that the fingerprint-matchingserver 106, the content-presentation device 104, or another entity canextract the channel identifier and use it to identify the channel onwhich the second content is being received by the content-presentationdevice 104.

In this context, the content-modification system 100 can employ anywatermark technique now known or later developed.

M. Operations Related to Concatenation of Multiple Supplemental ContentSegments

Content modification as described above by way of example illustratesvarious operations for modification of a single segment of content thathas been identified as modifiable, either by replacing the singlesegment with supplemental content, or by overlaying supplemental contenton the single segment. In either case, the modification operationentails presenting the supplemental content during a time intervaloriginally scheduled for presentation of the single segment of contentby the content-presentation device 104. In practice, there can becircumstances in which two or more segments of supplemental content maybe identified for consecutive, back-to-back content-modificationoperations by a given content-presentation device. Thus, instead ofmodification of a single segment of content, two or more consecutivesegments of content may be subject to two or more consecutivecontent-modification operations by a content-presentation device. Inorder to accommodate such situations with efficiency and in astreamlined fashion, further operations may be used to “stitch”together, or concatenate, two or more segments of supplemental contentso as to ensure contiguous download and delivery of multiplesupplemental content segments to the content-presentation device, and tofacilitate consecutive content-modification operations with the multiplesupplemental content segments.

Various operations relating to content “stitching” are now described. Byway of example, stitching operations are illustrated for the case of twoconsecutive supplemental content segments. The two supplemental contentsegments are referred below as “first supplemental content” and “secondsupplemental content.” Similarly, the content-modification operationscorresponding to the first and second supplemental content are referredto as a “first content-modification operation” and “secondcontent-modification operation.” It should be understood that theexample operations could be extended for more than two supplementalcontent segments, and that the illustration of two supplemental contentsegments is not limiting with respect to example embodiments describedherein.

Accordingly, in another aspect of the disclosure, the content-managementsystem 108 and the content-presentation device 104 can perform acts tofacilitate the content-presentation device 104 using a link to retrievefirst and second supplemental content, which the content-presentationdevice 104 can use in connection with performing first and secondcontent-modification operations, where the second content-modificationoperation is performed immediately after the first content-modificationoperation is performed. Examples of content stitching will first bedescribed in general terms, after which a more detailed description ispresented, also by way of example.

In one example configuration, the content-management system 108 canreceive from the content-presentation device 104, a request for firstsupplemental content for use by the content-presentation device 104 inconnection with performing a first content-modification operation.

The content-management system 108 can transmit to thecontent-presentation device 104, a link that points to a set of contentitems defined by a playlist, where the playlist includes firstsupplemental content. In one example, the transmitted link is associatedwith the content-presentation device 104 and is transmitted to only onecontent-presentation device, which is the content-presentation device104. In this way, the transmitted link can be a device-specific link.

The content-management system 108 can receive from thecontent-presentation device 104, a request for second supplementalcontent for use by the content-presentation device 104 in connectionwith performing a second content-modification operation.

The content-management system 106 can determine that thecontent-presentation device 104 is scheduled to perform the secondcontent-modification operation immediately after performing the firstcontent-modification operation. In one example, the content-managementsystem 108 can receive first metadata that specifies scheduling datarelated to the first content-modification operation and can receivesecond metadata that specifies scheduling data related to the secondcontent-modification operation. In this case, the content-managementsystem 108 determining that the content-presentation device 104 isscheduled to perform the second content-modification operationimmediately after performing the first content-modification operationcan involve the content-management system 108 using at least thereceived first metadata and the received second metadata to determinethat the content-presentation device 104 is scheduled to perform thesecond content-modification operation immediately after performing thefirst content-modification operation.

Responsive to determining that the content-presentation device 104 isscheduled to perform the second content-modification operationimmediately after performing the first content-modification operation,the content-management system 108 can cause the playlist to be modifiedsuch that that modified playlist further includes the secondsupplemental content positioned immediately after the first supplementalcontent. In one example, this can involve the content-management system108 appending or concatenating, within a memory buffer, the secondsupplemental content to the first supplemental content, therebystitching together the first supplemental content and the secondsupplement content together, in that order, within the memory buffer.

The content-management system 108 can cause the playlist to be modifiedas described above, to facilitate the content-presentation device 104(i) using the link to retrieve the first supplemental content and thesecond supplemental content, (ii) performing the firstcontent-modification operation, and (iii) performing the secondcontent-modification operation immediately after performing the firstcontent-modification operation. In some instances, thecontent-management system 108 can also transmit to thecontent-presentation device, a message indicating that the link pointsto the first supplemental content and the second supplemental content,to facilitate the content-presentation device 104 performing one or moreof the acts described above.

From the content-presentation device 104 perspective, thecontent-presentation device 104 can transmit a request for firstsupplemental content and can transmit a request for second supplementalcontent. The content-presentation device 104 can also receive a linkthat points to a set of content items defined by a playlist, wherein theplaylist includes the requested first supplemental content and therequested second supplemental content positioned immediately after thefirst supplemental content. The content-presentation device 104 can (i)use the received link to retrieve the first supplemental content and thesecond supplemental content, (ii) use the retrieved first supplementalcontent in connection with performing a first content-modificationoperation; and (iii) use the retrieved second supplemental content inconnection with performing a second content-modification operationimmediately after the first content-modification operation is performed.

In one example, the content-presentation device 104 can receive amessage indicating that the received link points to the firstsupplemental content and the second supplemental content. In this case,the content-presentation device 104 can perform one or more of the actsdescribed above, in response to receiving the message.

A more detail example of content stitching is illustrated in FIG. 5A,which shows certain example operations that may be carried out by thecontent-presentation device 104, the fingerprint-matching server 106,the content-management system 108, and the supplemental-content deliverysystem 112, for example. The example operations are represented in termsof a message flow or communication flow between the example systemelements. In FIG. 5A, time increases downward, thus setting a temporalorder of messages or communication between the example elements. Itshould be appreciated that the temporal spacing between messages is notnecessarily intended to indicate an actual amount of time, nor are therelative sizes of the spacings as illustrated in the figure intended toindicate actual relative sizes. Further, it should be noted that thespecific relative orderings of at least some of the messages areillustrative, and other orderings are possible.

For context, the messages and steps, labeled M1, . . . , M16, correspondapproximately to operations in T27-T41 of FIGS. 4D-4F described above,but with modifications or additions for implementation of contentstitching.

In accordance with example embodiments, the fingerprint-matching server106 may send message M1 to the content-presentation device 104 includingfingerprint and metadata of first supplemental content that has beenidentified as replacement content for broadcast content during a firstupcoming temporal interval. Message M1 may correspond to operations intime T27. In an example, the first supplemental content may also beidentified as specific to the content-presentation device 104.

In response to message M1, the content-presentation device 104 may senda message M2 to the content-management system 108 requesting the firstsupplemental content. The message may include metadata associated withthe first supplemental content, such that the content-management system108 may determine when the first supplemental content is scheduled to bepresented by the content-presentation device 104 as replacement content.Message M2 may correspond to operations in time T35.

In response to message M2, the content-management system 108 may send amessage M3 to the supplemental-content delivery system 112 requesting alink to a playlist that includes the first supplemental content. MessageM3 may correspond to operations in time T37.

Note that the term “playlist” as used in the present illustration may betaken to a computer-readable storage location containing actual content,possibly as well as associated metadata. This arrangement may differfrom some customary ones in which a playlist may contain only metadataand/or logical pointers to locations from which associated content maybe accessed or retrieved. It should be understood, however, that theexample operation described herein could be modified and/or extended towork with such customary playlist arrangements. For example, thesupplemental-content delivery system could be or include acontent-delivery network (CDN) in which actual content may be stored inmultiple, possibly distributed, locations. In this arrangement, aplaylist could be a list of pointers to addressable locations containingactual content associated with the pointers. However, the link to theplaylist may make details of the actual content storage transparent toan entity or device that may use the link to access and/or retrieve thecontent. In this sense, the link to the playlist may be consideredeffectively a link to the content represented in the playlist, whetherdirectly contained or indirectly pointed to.

In response to message M3, the supplemental-content delivery system 112may send a message M4 to the content-presentation device 104 including alink to the playlist, and thus to the first supplemental content.Although not necessarily shown, the content-presentation device 104 mayalso schedule a first content-modification operation for presentation ofthe first supplemental content. Message M3 may correspond approximatelyto operations in time T38. More particularly, in the operationsillustrated in FIGS. 4E and 4F, the link sent from thesupplemental-content delivery system 112 is received by thecontent-management system 108, which then sends the link to thecontent-presentation device. In the present illustration, thesupplemental-content delivery system 112 sends the link directly to thecontent-presentation device 104. This is an example of a possiblevariation of operations that may achieve the same ends.

Next, the fingerprint-matching server 106 may send message M5 to thecontent-presentation device 104 including fingerprint and metadata ofsecond supplemental content that has been identified as replacementcontent for broadcast content during a second upcoming temporal intervalthat immediately follows the first temporal interval. Message M5 maycorrespond to a second set of operations in time T27, this time relatingto the second supplemental content. In an example, the secondsupplemental content may also be identified as specific to thecontent-presentation device 104.

In response to message M5, the content-presentation device 104 may senda message M6 to the content-management system 108 requesting the secondsupplemental content. The message may include metadata associated withthe second supplemental content, such that the content-management system108 may determine when the second supplemental content is scheduled tobe presented by the content-presentation device 104 as replacementcontent. In particular, the content-management system 108 may determinethat the second supplemental content is scheduled to be presented by thecontent-presentation device 104 immediately following presentation ofthe first supplemental content. Message M6 may correspond to a secondset of operations in time T35, this time relating to the secondsupplemental content.

In response to receiving message M6, and determining that the secondsupplemental content is scheduled to be presented by thecontent-presentation device 104 immediately following presentation ofthe first supplemental content the content-management system 108, thecontent-management system 108 may send a message M7 to thesupplemental-content delivery system 122 instructing thesupplemental-content delivery system to stitch the second supplementalcontent to the end of the first supplemental content. In this way, thesupplemental-content delivery system 112 may cause the secondsupplemental content to be positioned immediately following the firstsupplemental content in the playlist. Since the link sent in message M4to the content-presentation device points to the playlist, the same linkmay now be used to retrieve both the first and second supplementalcontent from the supplemental-content delivery system.

In response to receiving message M7, the supplemental-content deliverysystem 112, may carry out operation M8, which entails concatenating, orstitching, the second supplemental content to the first supplementalcontent. The operation may be done in an internal buffer or other memorystorage of the supplemental-content delivery system used, for example,for staging download of the first and second supplemental content fromthe supplemental-content delivery system 112 to the content-presentationdevice 104. As such, the second supplemental content may be positionedimmediately following the first supplemental content in the internalbuffer. Further, the link pointing to the playlist may ultimately bepointing, directly or indirectly to the internal buffer.

Considering again various implementations of playlists, the operation ofstitching or concatenating the second supplemental content to the end ofthe first supplemental content could involve either relative positioningof actual content in memory or relative positioning of pointers inplaylist memory to actual content locations in content memory. In anexample implementation in which the playlist contains actual content,positioning the second supplemental content immediately after the firstsupplemental content in a buffer or other form of memory may entailpositioning the actual data making up the second supplemental contentimmediately after the actual data making up the first supplementalcontent. In an example implementation in which the playlist containspointers to the content locations containing the actual data of thefirst and second supplemental content, positioning the secondsupplemental content immediately after the first supplemental content ina buffer or other form of memory may entail positioning in the playlista pointer to the actual second supplemental content immediatelyfollowing a pointer to the actual first supplemental content. In thepointer implementation of the playlist, the actual locations of thefirst and second supplemental content could contiguous in memory, orseparate.

Having sent message M7 to the supplemental-content delivery system 112,the content-management system 108 may then send a message M9 to thecontent-presentation device 104 indicating or notifying that the linkpoints to both the first and second supplemental content. In the presentillustration, the content-presentation device 104 already obtained thelink in message M4.

In operation M10, the content-presentation device 104 schedules a secondpresentation-modification operation for presentation of the secondsupplemental content immediately following the firstcontent-modification operation. That is, the first and secondcontent-modification operation are scheduled back-to-back.

The content-presentation device 104 may then send a message M11including the link to the supplemental-content delivery system 112 in arequest to download the first supplemental content and the secondsupplemental content. In the present illustration, message M11 containsa single request for downloading both the first and second supplementalcontent. In an alternative operation described below in connection withFIG. 5B, a separate request may be sent for downloading the first andsecond supplemental content.

In response to message M11, the supplemental-content delivery system 112may start downloading, and the content-presentation device 104 may startreceiving, the first supplemental content in operation M12.

At some time after the download of the first supplemental contentbegins, the content-presentation device 104 may start the firstpresentation modification operation in operation M13. In the presentillustration, the first content-modification operation may be areplacement operation in which the first supplemental content is thereplacement content. In practice, the content-presentation device 104may receive a sufficient amount of the first supplemental content priorto the scheduled start time of the replacement operation, such that thefirst supplemental content is available for the complete contentreplacement of the first content-modification operation.

The supplemental-content delivery system 112 may start downloading, andthe content-presentation device 104 may start receiving, the secondsupplemental content in operation M14. Since the second supplementalcontent is concatenated to the end of the first supplemental content inthe playlist, the start of the download of the second supplementalcontent may be continuous with the completion of the download of thefirst supplemental content. Since M14 occurs prior to completion of thefirst content-modification operation, it may be seen that the firstsupplemental content is completely received in time to be fullyavailable for the replacement operation of the firstcontent-modification operation.

At some time after the download of the second supplemental contentbegins, the content-presentation device 104 may start the secondpresentation-modification operation in operation M15. In the presentillustration, the second content-modification operation may be areplacement operation in which the second supplemental content is thereplacement content. In the present illustration, thecontent-presentation device 104 may be assumed to receive a sufficientamount of the second supplemental content prior to the scheduled starttime of the replacement operation, such that the second supplementalcontent is available for the complete content replacement of the secondcontent-modification operation.

The download of the second supplemental content may complete atoperation M16. Thus, in the present illustration, the secondsupplemental content is completely received in time to be fullyavailable for the replacement operation of the secondcontent-modification operation. This timing circumstance may not alwaysbe guaranteed. Example operation discussed below in connection with FIG.5B address timing circumstance where this is not the case.

N. Operations Related to Unscheduling Content-modification operations ofConcatenated Supplemental Content Segments

The example content stitching operations described above apply, at leastimplicitly, to a timing circumstance in which the entirety of the secondsupplemental content is received by the content-presentation device 104in sufficient time to be available for completion of the secondpresentation modification operation. This is illustrated by message M15,which marks completion of the second supplemental content download priorto completion of the second presentation-modification operation, M16.That is, although the second presentation-modification operation beginsbefore the download of the second supplemental content is complete, thedownload evidently, and by way of example, completes in time forsuccessful performance of the second presentation-modificationoperation. However, this circumstance may not necessarily be guaranteed.For example, for a supplemental-content delivery system 112 implementedas a CDN or the like, transmission and/or network delays could causedownload to the content-presentation device of the second supplementalcontent to take too long to complete in time to be available for thecomplete second content-modification operation. The content-stitchingoperations as described in FIG. 5A alone may not necessarily accountsuch situations of excessive download delays.

In particular, if the second presentation-modification operation is afull segment replacement, and the entire second supplemental content isnot received by end of the segment being replaced, there can be a gapduring which no content at all is available for playout (presentation).To address this scenario, the content-presentation device 104 may carryout additional and/or modified operations that, prior to the completionof the first presentation-modification operation, compare the progressof the download of the second supplemental content with the scheduledstart time of the second content-modification start time. If less than athreshold portion of the first content-modification operation remains tobe completed while, at the same time, the download of the secondsupplemental content has not yet completed, then thecontent-presentation device 104 may unscheduled—or effectivelycancel—the second content-modification operation. In this instance, thecontent-presentation device 104 can revert to presenting the originallyscheduled content during the time slot that the secondcontent-modification operation had been scheduled in. Further, thecontent-presentation device 104 may modify playout (presentation) of theoriginally scheduled content with overlay content, if such overlaycontent is available and the timing requirements of the overlay contentcan be accommodated within the time slot.

Accordingly, in another aspect of the disclosure, thecontent-presentation device 104 can perform acts to facilitateunscheduling a content-modification operation. Examples of unschedulinga content-modification operation will first be described in generalterms, after which a more detailed description is presented, also by wayof example. A number of the operations involved may be the same orsimilar to operations of content stitching, particular those relating toreceiving first and second supplemental content, and scheduling firstand second content-modification operations. However, by way of example,some of the operations of unscheduling are described with a differentordering than similar operations of content stitching. These differencesat least illustrate variations that may be accommodated by and/or thatmay characterize different embodiments.

In one example configuration, the content-presentation device 104 canreceive a link that points to a set of content items defined by aplaylist, wherein when the link is received, the playlist includes firstsupplemental content, and wherein after the playlist is received, theplaylist is modified to include second supplemental content positionedimmediately after the first supplemental content. Thecontent-presentation device 104 can use the received link to retrievethe first supplemental content and can perform a firstcontent-modification operation that involves outputting for presentationthe retrieved first supplemental content.

Before completing performing the first content-modification operation,the content-presentation device 104 can (i) use the received link tostart retrieving the second supplemental content for use in connectionwith a second content-modification operation that thecontent-presentation device 104 is scheduled to perform immediatelyafter performing the first content-modification operation, and (ii)determine that while a less than a threshold portion of the firstcontent-modification operation remains to be performed, thecontent-presentation device 104 has not completed retrieving the secondsupplemental content. Responsive to determining this, thecontent-presentation device can unschedule the secondcontent-modification operation.

In one example, the second content-modification operation can be acontent replacement operation. In this example, the content-presentationdevice 104 unscheduling the second content-modification operation caninvolve the content-presentation device 104 replacing the scheduledsecond content-modification operation with a third content-modificationoperation that is a content overlay operation.

In some cases, the content-presentation device 104 can receive a messageindicating that the link points to the second supplemental content. Insuch cases, the content-presentation device 104 can determine that whileless than the threshold portion of the first content-modificationoperation remains to be performed, the content-presentation device 104has not completed retrieving the second supplemental content isperformed, responsive to receiving the message.

A more detail example of unscheduling a content-modification operationis illustrated in FIG. 5B, which shows certain example operations thatmay be carried out by the content-presentation device 104, for example.The format of FIG. 5B is the same as that of FIG. 5A. In particular,operations M1-M10 of content stitching shown if FIG. 5A are taken to thesame as in the unscheduling shown in FIG. 5B. For the sake of brevity inFIG. 5B, operations M2-M8 are omitted, represented instead by verticalellipses. The operations specific to unscheduling are then shown inoperations N11-N16, although some may be the same or similar to thestitching operations described above.

After scheduling in M10 the second content-modification operation inoperation for presentation of the second supplemental contentimmediately following the first content-modification operation, thecontent-presentation device 104 may then send a message N11 includingthe link to the supplemental-content delivery system 112 in a request todownload the first supplemental content. In the present illustration,message N11 contains a single request for downloading just the firstsupplemental content.

In response to message N11, the supplemental-content delivery system 112may start downloading, and the content-presentation device 104 may startreceiving, the first supplemental content in operation N12.

At some time after the download of the first supplemental contentbegins, the content-presentation device 104 may start the firstpresentation-modification operation in operation M13. In the presentillustration, the first content-modification operation may be areplacement operation in which the first supplemental content is thereplacement content. In practice, the content-presentation device 104may receive a sufficient amount of the first supplemental content priorto the scheduled start time of the replacement operation, such that thefirst supplemental content is available for the complete contentreplacement of the first content-modification operation.

Prior to completion of the first presentation-modification operation,the content-presentation device 104 may then send a message N14including the link to the supplemental-content delivery system 112 in arequest to download the second supplemental content. In the presentillustration, message N14 contains a single request for downloading justthe second supplemental content.

The supplemental-content delivery system 112 may start downloading, andthe content-presentation device 104 may start receiving, the secondsupplemental content in operation M15. Since the second supplementalcontent is concatenated to the end of the first supplemental content inthe playlist, the start of the download of the second supplementalcontent may be continuous with the completion of the download of thefirst supplemental content. Since M15 occurs prior to completion of thefirst content-modification operation, it may be seen that the firstsupplemental content is completely received in time to be fullyavailable for the replacement operation of the firstcontent-modification operation.

Prior to completion of the first presentation-modification operation,the content-presentation device 104 may perform operation N16 in whichit determines that less than a threshold portion of the firstpresentation-modification remains to be completed, while the download ofthe second supplemental content has not yet completed. This may indicatethat, although a portion of the second supplemental content has beenreceived, the entirety of the second supplemental content may not bereceived in time to be available for the complete secondcontent-modification operation. In response to this determination, thecontent-presentation device may unscheduled the secondcontent-modification operation. This action may effectively cancel thesecond content-modification operation in order to avoid or avert asituation in which the second supplemental content “runs out” before thesecond content-modification operation is complete.

Since completion of the first content-modification operation coincideswith the end of the temporal interval of the replacement operation ofthe first content-modification operation, the threshold portion used inthe determination could be a fraction of that temporal interval or aspecific amount of time. For example, the threshold could be 0.1 (or10%) of the temporal interval, or a value such a 5 or 10 seconds. Otherfractions and/or values could be used as well. The threshold test may beviewed as a check of how soon the first content-modification operationwill complete and the second content-modification operation, scheduledimmediately thereafter, will begin. If the download of the secondsupplemental content has not yet completed by the time there is lessthan the threshold portion remaining, it may be deemed that the downloadwill not complete prior to the end of the second temporal interval inwhich the second supplemental content is scheduled to be used asreplacement content. Thus, unscheduling the second content-modificationoperation can avoid a gap in available content for presentation thatwould otherwise occur during the second temporal interval.

Although not explicitly shown in FIG. 5B, the content-presentationdevice 104 may still modify content during the second temporal intervalusing an overlay operation. The may be possible, for example, if thecontent-presentation device has previously received and stored thirdsupplemental content that may be suitable for overlaying on originalcontent. Such an overlay operation may not be subject to the more stricttiming considerations of a replacement operation.

As illustrated by way of example, some of the operations of FIGS. 5A and5B that achieve roughly the same ends may differ. For example, in FIG.5A, message M11 includes a single request for download of both the firstand second supplemental content, while in FIG. 5B, separate requests,messages N11 and N14, are used for the two downloads. These differencesshould not viewed or taken as the cause or reason the different timingcircumstances of the download of the second supplemental content inFIGS. 5A and 5B. Rather, they are illustrative of variations of messageconstruction and order that could be used in either scenario.

IV. EXAMPLE METHODS

FIG. 6 is a flow chart of an example method of content modification withstitching of supplemental content, such as that described above. Theexample method 600 illustrates operations carried out on a client sideby the content-presentation device, for example. Example method 600could be implemented on and/or carried out by a computing system, suchas computing system 200 described above, which may be configured tocarry out content-presentation operations also described above. Anon-limiting example of such computing system include thecontent-presentation device 104, for example. In particular, the examplemethod 600 could be implemented on and/or carried out by an examplecontent-presentation device 104 that may include one or more processors,and a non-transitory computer-readable storage medium having storedthereon program instructions that, upon execution by the one orprocessors, cause the content-presentation device 104 to carry outoperations including operations of example method 600.

At block 602, the method 600 includes the content-presentation devicedetermining an upcoming first temporal interval and an immediatelyfollowing second temporal interval during both of which the presentationof the received content is modifiable. This determination may be madewhile the content-presentation device is receiving content forpresentation, and could correspond to the content-presentation deviceevaluating fingerprint and metadata received in messages M1 and M5 inFIG. 5A, for example.

At block 604, the method 600 includes the content-presentation devicetransmitting a first request for first supplemental content. This couldcorrespond to message M2 in FIG. 5A, for example.

At block 606, the method 600 includes the content-presentation devicetransmitting a second request for second supplemental content. Thiscould correspond to message M6 in FIG. 5A, for example.

At block 608, the method 600 includes the content-presentation devicereceiving a link that points to a set of content items arranged in aplaylist, where, when the first request is transmitted, the playlistincludes the requested first supplemental content, and at subsequenttime after the first request is transmitted the playlist is modified toalso include the requested second supplemental content positionedimmediately after the first supplemental content. This timing maycorrespond to messages M2-M8 in FIG. 5A, for example. More particularly,at the time the content-presentation device 104 sends message M2, noaction has yet been taken to concatenate the first and secondsupplemental content. However, in response to receiving the message M5,the content-management system 108 causes the supplemental-contentdelivery system 112 to stitch the second supplemental content to the endof the first supplemental content, thereby positioning the secondsupplemental content positioned immediately after the first supplementalcontent in the playlist.

At block 610, the method 600 includes the content-presentation deviceusing the received link to retrieve the first supplemental content. Thiscould correspond to message N11 in FIG. 5B, for example.

At block 612, the method 600 includes the content-presentation deviceusing the retrieved first supplemental content in a firstcontent-modification operation to modify presentation during the firsttemporal interval of the received content. This could correspond tooperation N13 in FIG. 5B, for example.

At block 614, the method 600 includes the content-presentation devicebeginning receiving download of the second supplemental content, andchecking or monitoring the progress of the download relative to anamount of time remaining in the first content-modification operation. Inparticular, prior to completion of the first content-modificationoperation the content-presentation device may (i) use the received linkto start retrieving the second supplemental content for upcoming use ina second content-modification operation scheduled immediately followingthe first content-modification operation in order to modify presentationduring the second temporal interval of the received content, and (ii)make a determination, by the content-presentation device, that whileless than a threshold portion of the first content-modificationoperation remains to be completed, the content-presentation device hasnot completed retrieving the second supplemental content. This maycorrespond to the first action of operation N16 in FIG. 5B, for example.

Finally, at block 616, the method 600 includes the content-presentationdevice responding to the determination in block 614 by unscheduling thesecond content-modification operation. This may correspond to the secondaction of operation N16 in FIG. 5B, for example.

In accordance with example embodiments, transmitting the second requestfor second supplemental content may entail transmitting the secondrequest at a time that is one of prior to receiving the link, or afterreceiving the link. The example of FIG. 5A illustrates transmitting thesecond request (message M6) after receiving the link (message M4).However, although not illustrated in FIG. 5A or 5B, thecontent-presentation device could transmit the first and second requestsin a single request prior to receiving the link.

In accordance with example embodiments, using the received link toretrieve the first supplemental content may entail thecontent-presentation device transmitting a first content requestincluding the link to a content server in a network, and receiving thefirst supplemental content from the server in a first download operationin response to the first content request. Similarly, using the receivedlink to start retrieving the second supplemental content may entail thecontent-presentation device transmitting a second content requestincluding the link to the content server, and receiving the secondsupplemental content from the content server in a second downloadoperation initiated by the content server in response to the secondcontent request. Further, the second content request may be transmittedin a single message that includes the first content request and thesecond content request, or in a separate message including the secondcontent request, following a first message that includes the firstcontent request. Transmitting a single request message could correspondto message M11 in FIG. 5A, for example; transmitting separate messagescould correspond to messages N11 and N 14 in FIG. 5B, for example.Receiving the first supplemental content could correspond to messagesM12 or N12 in FIG. 5A or 5B, for example. The second download operationinitiated by the content server in response to the second contentrequest could correspond to messages M14 or N15 in FIG. 5A or 5B, forexample. In an example embodiment, the content server could be thesupplemental-content delivery system 112.

In accordance with example embodiments, the first download operation andthe second download operation may be two parts of a single downloadoperation initiated by the content server in response to the singlemessage that includes the first content request and the second contentrequest. In this case, the content server may automatically begindownloading the second supplemental content as soon as download of thefirst supplemental content is complete. Thus, the content-presentationdevice may see the two downloads as a single continuous download.

In accordance with example embodiments, making the determination thatwhile less than the threshold portion of the first content-modificationoperation remains to be completed, the content-presentation device hasnot completed retrieving the second supplemental content may involvedetermining at less than a threshold amount of time prior to the end ofthe first temporal interval that the second download operation has notcompleted. That is, the content-presentation device may wait to make thedetermination until less than a threshold amount of time remains in thefirst content-modification operation.

In accordance with example embodiments, the example method 600 mayfurther entail the content-presentation device receiving a messageindicating that the link points to the second supplemental content.Accordingly, using the received link to start retrieving the secondsupplemental content may involve using the received link to startretrieving the second supplemental content responsive to receiving themessage. The message could correspond to message M9 in FIGS. 5A and 5B.

In accordance with example embodiments, the example method 600 mayfurther involve the content-presentation device receiving a messageindicating that the link points to the second supplemental content.Accordingly, making the determination that while less than the thresholdportion of the first content-modification operation remains to becompleted, the content-presentation device has not completed retrievingthe second supplemental content may involve making the determinationresponsive to receiving the message. That is, the operation to check theprogress of the download of the second supplemental content may be madein response to being made aware that the second download will becontinuous with the first.

In accordance with example embodiments, unscheduling the secondcontent-modification operation may entail refraining from performing thesecond content-modification operation.

In accordance with example embodiments, the second content-modificationoperation may be a content replacement operation. In this case,unscheduling the second content-modification operation may involvereplacing the scheduled second content-modification operation with athird content-modification operation that is a content overlayoperation.

V. EXAMPLE VARIATIONS

Although the examples and features described above have been describedin connection with specific entities and specific operations, inpractice, there are likely to be many instances of these entities andmany instances of these operations being performed, perhapscontemporaneously or simultaneously, on a large-scale basis. Indeed, inpractice, the content-modification system 100 is likely to include manycontent-distribution systems (each potentially transmitting content onmany channels) and many content-presentation devices, with some or allof the described operations being performed on a routine and repeatingbasis in connection with some or all of these entities.

In addition, although some of the operations described in thisdisclosure have been described as being performed by a particularentity, the operations can be performed by any entity, such as the otherentities described in this disclosure. Further, although the operationshave been recited in a particular order and/or in connection withexample temporal language, the operations need not be performed in theorder recited and need not be performed in accordance with anyparticular temporal restrictions. However, in some instances, it can bedesired to perform one or more of the operations in the order recited,in another order, and/or in a manner where at least some of theoperations are performed contemporaneously/simultaneously. Likewise, insome instances, it can be desired to perform one or more of theoperations in accordance with one more or the recited temporalrestrictions or with other timing restrictions. Further, each of thedescribed operations can be performed responsive to performance of oneor more of the other described operations. Also, not all of theoperations need to be performed to achieve one or more of the benefitsprovided by the disclosure, and therefore not all of the operations arerequired.

Although certain variations have been described in connection with oneor more examples of this disclosure, these variations can also beapplied to some or all of the other examples of this disclosure as welland therefore aspects of this disclosure can be combined and/or arrangedin many ways. The examples described in this disclosure were selected atleast in part because they help explain the practical application of thevarious described features.

Also, although select examples of this disclosure have been described,alterations and permutations of these examples will be apparent to thoseof ordinary skill in the art. Other changes, substitutions, and/oralterations are also possible without departing from the invention inits broader aspects as set forth in the following claims.

The invention claimed is:
 1. A method comprising: while receivingcontent for presentation, a content-presentation device determining anupcoming first temporal interval and an immediately following secondtemporal interval during both of which the presentation of the receivedcontent is modifiable; transmitting, by the content-presentation device,a first request for first supplemental content; transmitting, by thecontent-presentation device, a second request for second supplementalcontent; receiving, by the content-presentation device, a link thatpoints to a set of content items arranged in a playlist, wherein whenthe first request is transmitted, the playlist includes the requestedfirst supplemental content, and at subsequent time after the firstrequest is transmitted the playlist is modified to also include therequested second supplemental content positioned immediately after thefirst supplemental content; using, by the content-presentation device,the received link to retrieve the first supplemental content; using, bythe content-presentation device, the retrieved first supplementalcontent in a first content-modification operation to modify presentationduring the first temporal interval of the received content; prior tocompletion of the first content-modification operation: (i) using, bythe content-presentation device, the received link to start retrievingthe second supplemental content for upcoming use in a secondcontent-modification operation scheduled immediately following the firstcontent-modification operation in order to modify presentation duringthe second temporal interval of the received content, and (ii) making adetermination, by the content-presentation device, that while less thana threshold portion of the first content-modification operation remainsto be completed, the content-presentation device has not completedretrieving the second supplemental content; and responsive to thedetermination, unscheduling, by the content-presentation device, thesecond content-modification operation, wherein unscheduling the secondcontent-modification operation comprises: prior to the start of thesecond temporal interval, canceling modification of the received contentwith the second supplemental content that had been scheduled for thesecond temporal interval.
 2. The method of claim 1, wherein transmittingthe second request for second supplemental content comprisestransmitting the second request at a time that is one of prior toreceiving the link, or after receiving the link.
 3. The method of claim1, wherein using the received link to retrieve the first supplementalcontent comprises transmitting, by the content-presentation device, afirst content request to a content server in a network, the firstcontent request including the link, and receiving the first supplementalcontent from the content server in a first download operation inresponse to the first content request, and wherein using the receivedlink to start retrieving the second supplemental content comprisestransmitting, by the content-presentation device, a second contentrequest to the content server, the second content request including thelink, and receiving the second supplemental content from the contentserver in a second download operation initiated by the content server inresponse to the second content request, wherein the second contentrequest is transmitted in one of (i) a single message that includes thefirst content request and the second content request, or (ii) a separatemessage including the second content request, following a first messagethat includes the first content request.
 4. The method of claim 3,wherein the first download operation and the second download operationare two parts of a single download operation initiated by the contentserver in response to the single message that includes the first contentrequest and the second content request.
 5. The method of claim 3,wherein making the determination that while less than the thresholdportion of the first content-modification operation remains to becompleted, the content-presentation device has not completed retrievingthe second supplemental content comprises determining at less than athreshold amount of time prior to the end of the first temporal intervalthat the second download operation has not completed.
 6. The method ofclaim 1, further comprising: receiving, by the content-presentationdevice, a message indicating that the link points to the secondsupplemental content, wherein using the received link to startretrieving the second supplemental content comprises using the receivedlink to start retrieving the second supplemental content responsive toreceiving the message.
 7. The method of claim 1, further comprising:receiving, by the content-presentation device, a message indicating thatthe link points to the second supplemental content, wherein making thedetermination that while less than the threshold portion of the firstcontent-modification operation remains to be completed, thecontent-presentation device has not completed retrieving the secondsupplemental content comprises making the determination responsive toreceiving the message.
 8. The method of claim 1, wherein the secondcontent-modification operation is a content replacement operation, andwherein unscheduling the second content-modification operation furthercomprises replacing the scheduled second content-modification operationwith a third content-modification operation that is a content overlayoperation.
 9. A content-presentation device comprising: acontent-presentation element configured for presenting content receivedby the content-presentation device; one or more processors; and anon-transitory computer-readable storage medium having stored thereonprogram instructions that, upon execution by the one or processors,cause the content-presentation device to carry out operations including:while receiving content for presentation on the content-presentationelement, determining an upcoming first temporal interval and animmediately following second temporal interval during both of which thepresentation of the received content is modifiable; transmitting a firstrequest for first supplemental content; transmitting a second requestfor second supplemental content; receiving a link that points to a setof content items arranged in a playlist, wherein when the first requestis transmitted, the playlist includes the requested first supplementalcontent, and at subsequent time after the first request is transmittedthe playlist is modified to also include the requested secondsupplemental content positioned immediately after the first supplementalcontent; using the received link to retrieve the first supplementalcontent; using the retrieved first supplemental content in a firstcontent-modification operation to modify presentation during the firsttemporal interval of the received content; prior to completion of thefirst content-modification operation: (i) using the received link tostart retrieving the second supplemental content for upcoming use in asecond content-modification operation scheduled immediately followingthe first content-modification operation in order to modify presentationduring the second temporal interval of the received content, and (ii)making a determination that while less than a threshold portion of thefirst content-modification operation remains to be completed, retrievingthe second supplemental content has not yet completed; and responsive tothe determination, unscheduling the second content-modificationoperation, wherein unscheduling the second content-modificationoperation comprises: prior to the start of the second temporal interval,canceling modification of the received content with the secondsupplemental content that had been scheduled for the second temporalinterval.
 10. The content-presentation device of claim 9, whereintransmitting the second request for second supplemental contentcomprises transmitting the second request at a time that is one of priorto receiving the link, or after receiving the link.
 11. Thecontent-presentation device of claim 9, wherein using the received linkto retrieve the first supplemental content comprises transmitting afirst content request to a content server in a network, the firstcontent request including the link, and receiving the first supplementalcontent from the content server in a first download operation inresponse to the first content request, and wherein using the receivedlink to start retrieving the second supplemental content comprisestransmitting a second content request to the content server, the secondcontent request including the link, and receiving the secondsupplemental content from the content server in a second downloadoperation initiated by the content server in response to the secondcontent request, wherein the second content request is transmitted inone of (i) a single message that includes the first content request andthe second content request, or (ii) a separate message including thesecond content request, following a first message that includes thefirst content request.
 12. The content-presentation device of claim 11,wherein the first download operation and the second download operationare two parts of a single download operation initiated by the contentserver in response to the single message that includes the first contentrequest and the second content request.
 13. The content-presentationdevice of claim 11, wherein making the determination that while lessthan the threshold portion of the first content-modification operationremains to be completed, retrieving the second supplemental content hasnot yet completed comprises determining at less than a threshold amountof time prior to the end of the first temporal interval that the seconddownload operation has not completed.
 14. The content-presentationdevice of claim 9, wherein the operations further include: receiving amessage indicating that the link points to the second supplementalcontent, wherein using the received link to start retrieving the secondsupplemental content comprises using the received link to startretrieving the second supplemental content responsive to receiving themessage.
 15. The content-presentation device of claim 9, wherein theoperations further include: receiving a message indicating that the linkpoints to the second supplemental content, wherein making thedetermination that while less than the threshold portion of the firstcontent-modification operation remains to be completed, retrieving thesecond supplemental content has not yet completed comprises making thedetermination responsive to receiving the message.
 16. Thecontent-presentation device of claim 9, wherein the secondcontent-modification operation is a content replacement operation, andwherein unscheduling the second content-modification operation comprisesfurther replacing the scheduled second content-modification operationwith a third content-modification operation that is a content overlayoperation.
 17. A non-transitory computer-readable storage medium havingstored thereon program instructions that, upon execution by one or moreprocessors of a content-presentation device, cause thecontent-presentation device to carry out operations including: whilereceiving content for presentation determining an upcoming firsttemporal interval and an immediately following second temporal intervalduring both of which the presentation of the received content ismodifiable; transmitting a first request for first supplemental content;transmitting a second request for second supplemental content; receivinga link that points to a set of content items arranged in a playlist,wherein when the first request is transmitted, the playlist includes therequested first supplemental content, and at subsequent time after thefirst request is transmitted the playlist is modified to also includethe requested second supplemental content positioned immediately afterthe first supplemental content; using the received link to retrieve thefirst supplemental content; using the retrieved first supplementalcontent in a first content-modification operation to modify presentationduring the first temporal interval of the received content; prior tocompletion of the first content-modification operation: (i) using thereceived link to start retrieving the second supplemental content forupcoming use in a second content-modification operation scheduledimmediately following the first content-modification operation in orderto modify presentation during the second temporal interval of thereceived content, and (ii) making a determination that while less than athreshold portion of the first content-modification operation remains tobe completed, retrieving the second supplemental content has not yetcompleted; and responsive to the determination, unscheduling the secondcontent-modification operation wherein unscheduling the secondcontent-modification operation comprises: prior to the start of thesecond temporal interval, canceling modification of the received contentwith the second supplemental content that had been scheduled for thesecond temporal interval.
 18. The non-transitory computer-readablestorage medium of claim 17, wherein the second content-modificationoperation is a content replacement operation, and wherein unschedulingthe second content-modification operation further comprises replacingthe scheduled second content-modification operation with a thirdcontent-modification operation that is a content overlay operation.